Analyzing the Spatio-Temporal Domain: from View Synthesis to Motion Segmentation

In this work I investigate spatio-temporal information in a video sequence. The advantage of considering a video sequence as a 3D spatio-temporal function with temporal continuity (rather than merely a discrete collection of 2D images) is demonstrated by two computer vision techniques which I have developed. View Synthesis: Each frame of the video sequence is an intersection of the spatio-temporal video volume with a spatial plane. When a video sequence conforms to certain geometrical constraints, intersecting the video volume with other planes or surfaces can be used to easily produce new views of the scene. This powerful view synthesis technique is based solely on captured data and does not require scene reconstruction, as the constraint on the input camera motion make it invariant to the scene structure in some respects. This technique is demonstrated with real sequences, giving visually appealing results. The technique gives rise to a novel projection model, the Crossed-Slits projection, that can be seen as a generalization of the perspective projection and several other models. A Crossed-Slits camera is defined by two lines which all rays must intersect. Here I study this new projection model and its epipolar geometry, which are shown to be quadratic equivalents of the perspective model. Crossed-Slits images are not perspective, and thus they appear distorted. These distortions are studied, and two frameworks are developed for handling them: First, assuming that a coarse approximation of the scene structure is known (which is used to create a realtime omnidirectional virtual environment); Second, without any knowledge about the scene, based only on the set of rays. In both cases distortion is reduced by approximating the perspective projection. iii The work on view synthesis and the Crossed-Slits projection, presented in Chapter 3 and 4, is based on work published in [1–6]. Motion Segmentation: Analysis of an unconstrained video sequence in general motion reveals a highly regular spatio-temporal structure, where moving objects appear as continuous structures in the temporal domain, broken by occlusion. Based on this observation, I developed a novel motion segmentation algorithm from a video sequence in general motion, which is based on differential properties in the spatio-temporal domain. I present a differential occlusion detector, which detects corner-like features that are indicative of motion boundaries. Segmentation is achieved by integrating the response of this detector in scale space. The algorithm is shown to give good results on real sequences taken in general motion. Experiments with synthetic data show robustness to high levels of noise and illumination changes; the experiments also include cases where no intensity edge exists at the location of the motion boundary, or when no parametric motion model can describe the data Next I describe two algorithms to determine depth ordering from twoand three-frame sequences based on observations about the scale space characteristics of the motion boundary. An interesting property of this method is its ability compute depth ordering from only two frames, even when no edge can be detected in a single frame. Finally, experiments show that people, like my algorithm, can compute depth ordering from only two frames, even when the boundary between the layers is not visible in a single frame. The work on motion segmentation and depth ordering, presented in Chapter 5, is based on [7, 8].